Near-eye microlens array displays

ABSTRACT

In embodiments of the invention, an apparatus may include a display comprising a plurality of pixels and a computer system coupled with the display and operable to instruct the display to display images. The apparatus may further include a microlens array located adjacent to the display and comprising a plurality of microlenses, wherein the microlens array is operable to produce a light field by altering light emitted by the display to simulate an object that is in focus to an observer while the display and the microlens array are located within a near-eye range of the observer.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional Application No.61/667,362, filed Jul. 2, 2012, the entire disclosure of which isincorporated herein by reference. This application claims priority fromU.S. Provisional Application No. 61/668,953, filed Jul. 6, 2012, theentire disclosure of which is incorporated herein by reference. Thefollowing U.S. Patent Application are incorporated herein by referencefor all purposes: U.S. patent application Ser. No. 13/720,831, “NEAR-EYEPARALLAX BARRIER DISPLAYS,” David Luebke, filed Dec. 20, 2012; and U.S.patent application Ser. No. 13/720,842, “NEAR-EYE OPTICAL DECONVOLUTIONDISPLAYS,” David Luebke, filed Dec. 20, 2012.

BACKGROUND OF THE INVENTION

Near-eye displays (NEDs) include head-mounted displays (HMDs) that mayproject images directly into a viewer's eyes. Such displays may overcomethe limited screen size afforded by other mobile display form factors bysynthesizing virtual large-format display surfaces, or may be used forvirtual or augmented reality applications.

Near-eye displays can be divided into two broad categories: immersivedisplays and see-through displays. The former may be employed in virtualreality (VR) environments to completely encompass a user's field of viewwith synthetically-rendered imagery. The latter may be employed inaugmented reality (AR) applications, where text, other syntheticannotations, or images may be overlaid in a user's view of the physicalenvironment. In terms of display technology, AR applications requiresemi-transparent displays (e.g., achieved by optical or electro-opticalapproaches), such that the physical world may be viewed simultaneouslywith the near-eye display.

Near-eye displays have proven difficult to construct due to the factthat the unaided human eye cannot accommodate (focus) on objects placedwithin close distances, for example, the distance between the lenses ofreading glasses to a user's eye when the user is wearing the glasses. Asa result, NED systems have conventionally required complex and bulkyoptical elements to allow the viewer to comfortably accommodate on thenear-eye display, which would otherwise be out of focus, and thephysical environment.

A conventional solution is to place a beam-splitter (e.g., apartially-silvered mirror) directly in front of the viewer's eye. Thisallows a direct view of the physical scene, albeit with reducedbrightness. In addition, a display (e.g., an LCD panel) is placed on thesecondary optical path. Introducing a lens between the beam-splitter andthe display has the effect of synthesizing a semi-transparent displaylocated within the physical environment. In practice, multiple opticalelements are required to minimize aberrations and achieve a wide fieldof view for such a solution, leading to bulky and expensive eyewear thathas prohibited widespread consumer adoption.

A conventional solution for VR applications is to place a magnifier infront of a microdisplay. For example, a single lens placed over a smallLCD panel so that the viewer can both accommodate or focus on thedisplay, despite the close distance, as well as magnify the display, sothat it appears to be much larger and at a greater distance.

BRIEF SUMMARY OF THE INVENTION

In embodiments of the invention, an apparatus may include a displaycomprising a plurality of pixels and a computer system coupled with thedisplay and operable to instruct the display to display images. Theapparatus may further include a microlens array located adjacent to thedisplay and comprising a plurality of microlenses, wherein the microlensarray is operable to produce a light field by altering light emitted bythe display to simulate an object that is in focus to an observer whilethe display and the microlens array are located within a near-eye rangeof the observer.

Various embodiments of the invention may include an apparatus comprisinga display operable to produce an image. The apparatus may furtherinclude a microlens array located adjacent to the display, wherein themicrolens array together with the display is operable to produce a lightfield simulating a 3D object that is recognizable to an observer whilethe display and the microlens array are located within a near-eye rangeof the observer.

Some embodiments of the invention may include a method comprisingdetermining a pre-filtered image to be displayed, wherein thepre-filtered image corresponds to a target image. The method may furtherinclude displaying the pre-filtered image on a display and producing anear-eye light field after the pre-filtered image travels through amicrolens array adjacent to the display, wherein the near-eye lightfield is operable to simulate a light field corresponding to the targetimage.

The following detailed description together with the accompanyingdrawings will provide a better understanding of the nature andadvantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are illustrated by way of example,and not by way of limitation, in the figures of the accompanyingdrawings and in which like reference numerals refer to similar elements.

FIG. 1 is an exemplary computer system, in accordance with embodimentsof the present invention.

FIG. 2A illustrates an eye of an observer and a corresponding minimumaccommodation distance.

FIGS. 2B and 2C depict perceived images at different viewing distancesof an observer.

FIG. 3A illustrates a ray of light originating from a plane of focus,according to embodiments of the present invention.

FIG. 3B illustrates a side view of a near-eye microlens array display,according to embodiments of the present invention.

FIG. 4 illustrates a ray of light that is part of a light field,according to embodiments of the present invention.

FIG. 5 illustrates a side view of the magnified view of the near-eyemicrolens array display, according to embodiments of the presentinvention.

FIG. 6A illustrates a side view of a near-eye parallax barrier display,according to embodiments of the present invention.

FIG. 6B illustrates a side view of a near-eye parallax barrier displayand a microlens array, according to embodiments of the presentinvention.

FIG. 7 illustrates a magnified side view of the near-eye parallaxbarrier display, according to embodiments of the present invention.

FIG. 8 illustrates a side view of a near-eye multilayer SLM display,according to embodiments of the present invention.

FIG. 9 illustrates a magnified side view of the near-eye multilayer SLMdisplay, according to embodiments of the present invention.

FIG. 10 depicts a view through the near-eye parallax barrier display,according to embodiments of the present invention.

FIG. 11 illustrates a side view of a near-eye optical deconvolutiondisplay, according to embodiments of the present invention.

FIG. 12A depicts images before and after convolution, according toembodiments of the present invention.

FIG. 12B depicts images before and after deconvolution, according toembodiments of the present invention.

FIG. 12C depicts a deconvolved image before and after convolution,according to embodiments of the present invention.

FIG. 13 depicts a flowchart of an exemplary process of displaying anear-eye image, according to an embodiment of the present invention.

FIG. 14 depicts a flowchart of an exemplary process of displaying anear-eye image, according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the various embodiments of thepresent disclosure, examples of which are illustrated in theaccompanying drawings. While described in conjunction with theseembodiments, it will be understood that they are not intended to limitthe disclosure to these embodiments. On the contrary, the disclosure isintended to cover alternatives, modifications and equivalents, which maybe included within the spirit and scope of the disclosure as defined bythe appended claims. Furthermore, in the following detailed descriptionof the present disclosure, numerous specific details are set forth inorder to provide a thorough understanding of the present disclosure.However, it will be understood that the present disclosure may bepracticed without these specific details. In other instances, well-knownmethods, procedures, components, and circuits have not been described indetail so as not to unnecessarily obscure aspects of the presentdisclosure.

Some portions of the detailed descriptions that follow are presented interms of procedures, logic blocks, processing, and other symbolicrepresentations of operations on data bits within a computer memory.These descriptions and representations are the means used by thoseskilled in the data processing arts to most effectively convey thesubstance of their work to others skilled in the art. In the presentapplication, a procedure, logic block, process, or the like, isconceived to be a self-consistent sequence of steps or instructionsleading to a desired result. The steps are those utilizing physicalmanipulations of physical quantities. Usually, although not necessarily,these quantities take the form of electrical or magnetic signals capableof being stored, transferred, combined, compared, and otherwisemanipulated in a computer system. It has proven convenient at times,principally for reasons of common usage, to refer to these signals astransactions, bits, values, elements, symbols, characters, samples,pixels, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the followingdiscussions, it is appreciated that throughout the present disclosure,discussions utilizing terms such as “displaying,” “generating,”“producing,” “calculating,” “determining,” “radiating,” “emitting,”“attenuating,” “modulating,” “convoluting,” “deconvoluting,”“performing,” or the like, refer to actions and processes (e.g.,flowcharts 1300 and 1400 of FIGS. 13 and 14) of a computer system orsimilar electronic computing device or processor (e.g., system 110 ofFIG. 1). The computer system or similar electronic computing devicemanipulates and transforms data represented as physical (electronic)quantities within the computer system memories, registers or other suchinformation storage, transmission or display devices.

Embodiments described herein may be discussed in the general context ofcomputer-executable instructions residing on some form ofcomputer-readable storage medium, such as program modules, executed byone or more computers or other devices. By way of example, and notlimitation, computer-readable storage media may comprise non-transitorycomputer-readable storage media and communication media; non-transitorycomputer-readable media include all computer-readable media except for atransitory, propagating signal. Generally, program modules includeroutines, programs, objects, components, data structures, etc., thatperform particular tasks or implement particular abstract data types.The functionality of the program modules may be combined or distributedas desired in various embodiments.

Computer storage media includes volatile and nonvolatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer-readable instructions, data structures,program modules or other data. Computer storage media includes, but isnot limited to, random access memory (RAM), read only memory (ROM),electrically erasable programmable ROM (EEPROM), flash memory or othermemory technology, compact disk ROM (CD-ROM), digital versatile disks(DVDs) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium that can be used to store the desired information and that canaccessed to retrieve that information.

Communication media can embody computer-executable instructions, datastructures, and program modules, and includes any information deliverymedia. By way of example, and not limitation, communication mediaincludes wired media such as a wired network or direct-wired connection,and wireless media such as acoustic, radio frequency (RF), infrared, andother wireless media. Combinations of any of the above can also beincluded within the scope of computer-readable media.

FIG. 1 is a block diagram of an example of a computing system 110capable of implementing embodiments of the present disclosure. Computingsystem 110 broadly represents any single or multi-processor computingdevice or system capable of executing computer-readable instructions.Examples of computing system 110 include, without limitation,workstations, laptops, client-side terminals, servers, distributedcomputing systems, handheld devices, worn devices (e.g., head-mounted orwaist-worn devices), or any other computing system or device. In itsmost basic configuration, computing system 110 may include at least oneprocessor 114 and a system memory 116.

Processor 114 generally represents any type or form of processing unitcapable of processing data or interpreting and executing instructions.In certain embodiments, processor 114 may receive instructions from asoftware application or module. These instructions may cause processor114 to perform the functions of one or more of the example embodimentsdescribed and/or illustrated herein.

System memory 116 generally represents any type or form of volatile ornon-volatile storage device or medium capable of storing data and/orother computer-readable instructions. Examples of system memory 116include, without limitation, RAM, ROM, flash memory, or any othersuitable memory device. Although not required, in certain embodimentscomputing system 110 may include both a volatile memory unit (such as,for example, system memory 116) and a non-volatile storage device (suchas, for example, primary storage device 132).

Computing system 110 may also include one or more components or elementsin addition to processor 114 and system memory 116. For example, in theembodiment of FIG. 1, computing system 110 includes a memory controller118, an input/output (I/O) controller 120, and a communication interface122, each of which may be interconnected via a communicationinfrastructure 112. Communication infrastructure 112 generallyrepresents any type or form of infrastructure capable of facilitatingcommunication between one or more components of a computing device.Examples of communication infrastructure 112 include, withoutlimitation, a communication bus (such as an Industry StandardArchitecture (ISA), Peripheral Component Interconnect (PCI), PCI Express(PCIe), or similar bus) and a network.

Memory controller 118 generally represents any type or form of devicecapable of handling memory or data or controlling communication betweenone or more components of computing system 110. For example, memorycontroller 118 may control communication between processor 114, systemmemory 116, and I/O controller 120 via communication infrastructure 112.

I/O controller 120 generally represents any type or form of modulecapable of coordinating and/or controlling the input and outputfunctions of a computing device. For example, I/O controller 120 maycontrol or facilitate transfer of data between one or more elements ofcomputing system 110, such as processor 114, system memory 116,communication interface 122, display adapter 126, input interface 130,and storage interface 134.

Communication interface 122 broadly represents any type or form ofcommunication device or adapter capable of facilitating communicationbetween example computing system 110 and one or more additional devices.For example, communication interface 122 may facilitate communicationbetween computing system 110 and a private or public network includingadditional computing systems. Examples of communication interface 122include, without limitation, a wired network interface (such as anetwork interface card), a wireless network interface (such as awireless network interface card), a modem, and any other suitableinterface. In one embodiment, communication interface 122 provides adirect connection to a remote server via a direct link to a network,such as the Internet. Communication interface 122 may also indirectlyprovide such a connection through any other suitable connection.

Communication interface 122 may also represent a host adapter configuredto facilitate communication between computing system 110 and one or moreadditional network or storage devices via an external bus orcommunications channel. Examples of host adapters include, withoutlimitation, Small Computer System Interface (SCSI) host adapters,Universal Serial Bus (USB) host adapters, IEEE (Institute of Electricaland Electronics Engineers) 1394 host adapters, Serial AdvancedTechnology Attachment (SATA) and External SATA (eSATA) host adapters,Advanced Technology Attachment (ATA) and Parallel ATA (PATA) hostadapters, Fibre Channel interface adapters, Ethernet adapters, or thelike. Communication interface 122 may also allow computing system 110 toengage in distributed or remote computing. For example, communicationinterface 122 may receive instructions from a remote device or sendinstructions to a remote device for execution.

As illustrated in FIG. 1, computing system 110 may also include at leastone display device 124 coupled to communication infrastructure 112 via adisplay adapter 126. Display device 124 generally represents any type orform of device capable of visually displaying information forwarded bydisplay adapter 126. Similarly, display adapter 126 generally representsany type or form of device configured to forward graphics, text, andother data for display on display device 124.

As illustrated in FIG. 1, computing system 110 may also include at leastone input device 128 coupled to communication infrastructure 112 via aninput interface 130. Input device 128 generally represents any type orform of input device capable of providing input, either computer- orhuman-generated, to computing system 110. Examples of input device 128include, without limitation, a keyboard, a pointing device, a speechrecognition device, an eye-track adjustment system, environmentalmotion-tracking sensor, an internal motion-tracking sensor, a gyroscopicsensor, accelerometer sensor, an electronic compass sensor, or any otherinput device.

As illustrated in FIG. 1, computing system 110 may also include aprimary storage device 132 and a backup storage device 133 coupled tocommunication infrastructure 112 via a storage interface 134. Storagedevices 132 and 133 generally represent any type or form of storagedevice or medium capable of storing data and/or other computer-readableinstructions. For example, storage devices 132 and 133 may be a magneticdisk drive (e.g., a so-called hard drive), a floppy disk drive, amagnetic tape drive, an optical disk drive, a flash drive, or the like.Storage interface 134 generally represents any type or form of interfaceor device for transferring data between storage devices 132 and 133 andother components of computing system 110.

In one example, databases 140 may be stored in primary storage device132. Databases 140 may represent portions of a single database orcomputing device or it may represent multiple databases or computingdevices. For example, databases 140 may represent (be stored on) aportion of computing system 110 and/or portions of example networkarchitecture 200 in FIG. 2 (below). Alternatively, databases 140 mayrepresent (be stored on) one or more physically separate devices capableof being accessed by a computing device, such as computing system 110and/or portions of network architecture 200.

Continuing with reference to FIG. 1, storage devices 132 and 133 may beconfigured to read from and/or write to a removable storage unitconfigured to store computer software, data, or other computer-readableinformation. Examples of suitable removable storage units include,without limitation, a floppy disk, a magnetic tape, an optical disk, aflash memory device, or the like. Storage devices 132 and 133 may alsoinclude other similar structures or devices for allowing computersoftware, data, or other computer-readable instructions to be loadedinto computing system 110. For example, storage devices 132 and 133 maybe configured to read and write software, data, or othercomputer-readable information. Storage devices 132 and 133 may also be apart of computing system 110 or may be separate devices accessed throughother interface systems.

Many other devices or subsystems may be connected to computing system110. Conversely, all of the components and devices illustrated in FIG. 1need not be present to practice the embodiments described herein. Thedevices and subsystems referenced above may also be interconnected indifferent ways from that shown in FIG. 1. Computing system 110 may alsoemploy any number of software, firmware, and/or hardware configurations.For example, the example embodiments disclosed herein may be encoded asa computer program (also referred to as computer software, softwareapplications, computer-readable instructions, or computer control logic)on a computer-readable medium.

The computer-readable medium containing the computer program may beloaded into computing system 110. All or a portion of the computerprogram stored on the computer-readable medium may then be stored insystem memory 116 and/or various portions of storage devices 132 and133. When executed by processor 114, a computer program loaded intocomputing system 110 may cause processor 114 to perform and/or be ameans for performing the functions of the example embodiments describedand/or illustrated herein. Additionally or alternatively, the exampleembodiments described and/or illustrated herein may be implemented infirmware and/or hardware.

For example, a computer program for determining a pre-filtered imagebased on a target image may be stored on the computer-readable mediumand then stored in system memory 116 and/or various portions of storagedevices 132 and 133. When executed by the processor 114, the computerprogram may cause the processor 114 to perform and/or be a means forperforming the functions required for carrying out the determination ofa pre-filtered image discussed above.

Near-Eye Displays

Embodiments of the present invention provide near-eye displays includingthin stacks of semi-transparent displays operable to be placed directlyin front of a viewer's eye together with pre-processing algorithms forevaluating the depicted multilayer imagery, without the need foradditional costly or bulky optical elements to support comfortableaccommodation.

Embodiments of the present invention allow for attenuation-based lightfield displays that may allow lightweight near-eye displays. It shouldbe appreciated that other embodiments are not limited to onlyattenuation-based light field displays, but also light-emitting-basedlight field displays. Using near-eye light field displays, comfortableviewing may be achieved by synthesizing a light field corresponding to avirtual display located within the accommodation range of an observer.

Embodiments of the present invention provide near-eye displays includingone or more displays placed proximate to a viewer's eye where the targetimagery is deconvolved by the estimated point spread function for theeye, rather than synthesizing a light field supporting comfortableaccommodation. Further, embodiments of the present invention provideadditional methods for near-eye displays, including methods combininglight field display and optical deconvolution, as well as extensions toholographic displays.

FIG. 2A illustrates an eye 204 of an observer and a correspondingminimum accommodation distance 218. The eye 204 includes a lens 208 thatfocuses viewed objects onto a retina plane 212 of the eye 204. The eye204 may be capable of focusing on objects at various distances from theeye 204 and lens 208. For example, the eye 204 may be able to focus onan object that is located farther from the eye 204 than a near plane216, e.g., at a plane of focus 214 beyond the near plane 216.

Accordingly, the eye 204 may have a minimum accommodation distance 218that defines the minimum distance of an object at which the eye 204 iscapable of focusing on. In other words, the eye 204 may be incapable offocusing on an object that is located at a distance from the eye 204that is less than the minimum accommodation distance 218 or closer tothe eye 204 than the near plane 216. For example, if the surface of anobject is located at a near-eye plane 222 that is located a distancefrom the eye 204 less than the minimum accommodation distance 218, thesurface of the object will be out of focus to the observer. Objects thatare farther from the eye 204 than the near plane 216 are inside anaccommodation range and objects that are nearer to the eye 204 than thenear plane 216 are outside the accommodation range. Objects that arenearer to the eye 204 than the near plane 216 are in a near-eye range.

FIGS. 2B and 2C depict perceived images 230 and 240 at different viewingdistances of an observer. For example, FIG. 2B shows an eye exam chart230 as it would be perceived by an observer if it were located at theplane of focus 214 of the eye 204 in FIG. 2A. Or, the eye exam chart 230may be located at a different plane of focus, as long as the eye examchart 230 is within the accommodation range. As can be appreciated, theeye exam chart 230 is in focus, sharp, and/or recognizable.

Alternatively, FIG. 2C shows an eye exam chart 240 as it would beperceived by an observer if it were located nearer to the eye 204 thanthe plane of focus 214 in FIG. 2A. In other words, the eye exam chart230 may be located outside the accommodation range at, for example, thenear-eye plane 222. As can be appreciated, the eye exam chart 240 is outof focus, blurry, and/or unrecognizable.

Near-Eye Microlens Array Displays

Conventional displays, such as liquid crystal displays (LCDs) andorganic light-emitting diodes (OLEDs), may be designed to emit lightisotropically (uniformly) in all directions. In contrast, light fielddisplays support the control of individual rays of light. For example,the radiance of a ray of light may be modulated as a function ofposition across the display, as well as the direction in which the rayof light leaves the display.

FIG. 3A illustrates a ray of light 320 originating from a plane of focus214, according to embodiments of the present invention. FIG. 3A includesthe same eye 204, lens 208, retina plane 212, plane of focus 214, andaccommodation distance 218 of FIG. 2A. FIG. 3A also includes a ray oflight 320 that originates from the surface of an object that is locatedat the plane of focus 214. The origination point, angle, intensity, andcolor of the ray of light 320 and other rays of light viewable by theobserver provide a view of an in-focus object to the observer.

FIG. 3B illustrates a side view of a near-eye microlens array display301, according to embodiments of the present invention. FIG. 3B includesthe same elements as FIG. 3A, with the addition of a display 324 and amicrolens array 328. While FIG. 3A shows the microlens array 328 betweenthe display 324 and the eye 204, embodiments allow for the display 324to be positioned between the microlens array 328 and the eye 204.

The display 324 may be, but is not limited to being, an LCD or OLED. Themicrolens array 328 may be a collection of multiple microlenses. Themicrolens array 328 or each individual microlens may be formed bymultiple surfaces to minimize optical aberrations. The display 324 mayprovide an image, where the image emits rays of light isotropically.However, when the rays of light reach the microlens array 328, themicrolens array 328 may allow certain rays of light to refract toward orpass through toward the eye 204 while refracting other rays of lightaway from the eye 204.

Accordingly, the microlens array 328 may allow the light from selectpixels of the display 324 to refract toward or pass through toward theeye 204, while other rays of light pass through but refract away fromthe eye 204. As a result, the microlens array 328 may allow a ray oflight 321 to pass through, simulating the ray of light 320 of FIG. 3A.For example, the ray of light 321 may have the same angle, intensity,and color of the ray of light 320. Importantly, the ray of light 321does not have the same origination point as the ray of light 320 sinceit originates from display 324 and not the plane of focus 214, but fromthe perspective of the eye 204, the ray of light 320 is equivalent tothe ray of light 321. Therefore, regardless of the origination point ofthe ray of light 321, the object represented by the ray of light 321appears to be located at the plane of focus 214, when no object in factexists at the plane of focus 214.

It should be appreciated that the microlenses or the microlens array 328entirely may be electro-optically switchable such that the microlensarray 328 may be configured to be either transparent or opaque (e.g.,appearing as a flat sheet of glass). For example, the microlens array328 may be formed by liquid crystals or by birefringent optics, togetherwith polarizers. As a result, such switchable microlenses may beelectronically controlled, alternatingly from a microlens array operableto display a light field to an opaque element appearing similar to aflat sheet of glass, operable to allow the viewing of the surroundingenvironment. The transparent and opaque modes may be rapidly alternatedbetween, spatially-multiplexed, or combined spatially and temporallymodulated. Accordingly, augmented-reality applications may be provided,similar to those discussed with respect to FIGS. 6-10. Further,virtual-reality applications may be provided using a fixed microlensarray.

Importantly, the display 324 is located outside the accommodation rangeof the eye 204. In other words, the display 324 is located at a distanceless than the minimum accommodation distance 218. However, because themicrolens array 328 creates a light field (as discussed below) thatmimics or simulates the rays of light emitted by an object outside theminimum accommodation distance 218 that can be focused on, the imageshown by display 324 may be in focus.

FIG. 4 illustrates a ray of light 408 that is part of a light field,according to embodiments of the present invention. The light field maydefine or describe the appearance of a surface 404, multiplesuperimposed surfaces, or a general 3D scene. For a general virtual 3Dscene, the set of (virtual) rays that may impinge on the microlens array328 must be recreated by the near-eye display device. As a result, thesurface 404 would correspond to the plane of the display 324 and eachray 408 would correspond to a ray 320 intersecting the plane of thedisplay 324, resulting in the creation of an emitted ray 321 from thenear-eye light field display.

More specifically, the light field may include information for rays oflight for every point and light ray radiation angle on the surface 404,which may describe the appearance of the surface 404 from differentdistances and angles. For example, for every point on surface 404, andfor every radiation angle of a ray of light, information such asintensity and color of the ray of light may define a light field thatdescribes the appearance of the surface 404. Such information for eachpoint and radiation angle constitute the light field.

In FIG. 4, the ray of light 408 my radiate from an origination point 412of the surface 404, which may be described by an ‘x’ and ‘y’ coordinate.Further, the ray of light 408 may radiate into 3-dimensional space withan x (horizontal), y (vertical), and z (depth into and out of the page)component. Such an angle may be described by the angles Φ and θ.Therefore, each (x, y, Φ, θ) coordinate may describe a ray of light,e.g., the ray of light 408 shown. Each (x, y, Φ, θ) coordinate maycorrespond to a ray of light intensity and color, which together formthe light field. For video applications, the light field intensity andcolor may vary over time (t) as well.

Once the light field is known for the surface 404, the appearance of thesurface 404, with the absence of the actual surface 404, may be createdor simulated to an observer. The origination points of rays of lightsimulating the surface 404 may be different from the actual originationpoints of the actual rays of light from the surface 404, but from theperspective of an observer, the surface 404 may appear to exist as ifthe observer were actually viewing it.

Returning to FIG. 3B, the display 324 in conjunction with the microlensarray 328 may produce a light field that may mimic or simulate an objectat the plane of focus 214. As discussed above, from the perspective ofthe eye 204, the ray of light 321 may be equivalent to the ray of light320 of FIG. 3A. Therefore, an object that is simulated to be located atthe viewing plane 214 by the display 324 and the microlens array 328 mayappear to be in focus to the eye 204 because the equivalent light fieldfor a real object is simulated. Further, because the equivalent lightfield for a real object is simulated, the simulated object will appearto be 3-dimensional.

In some cases, limitations of a light field display's resolution maycause a produced ray of light to only approximately replicate ray. Forexample, with respect to FIGS. 3A and 3B, the ray of light 321 may havea slightly different color, intensity, position, or angle than the rayof light 320. Given the quality of the pre-filtering algorithm, thecapabilities of the near-eye light field display, and the ability of thehuman visual system to perceive differences, the set of rays 321 emittedby the near-eye display may approximate or fully replicate theappearance of a virtual object, such as the place 404. In cases wherethe appearance is approximated, rays may not need to be exactlyreplicated for appropriate or satisfactory image recognition.

FIG. 5 illustrates a magnified side view of the display 324 andmicrolens array 328 of FIG. 3B, according to embodiments of the presentinvention. FIG. 5 also includes the eye 204 of an observer of FIG. 3B.

The display 324 may include multiple pixels, for example, pixels 512,522, 524, and 532. There may be pixel groups, for example, the pixelgroup 510 including the pixel 512, the pixel group 520 including thepixels 522 and 524, and the pixel group 530 including the pixel 532.Each pixel group may correspond with a microlens of the microlens array328. For example, the pixel groups 510, 520, and 530 may be locatedadjacent to microlenses 516, 526, and 536, respectively.

As discussed above, the pixels may emit light isotropically (uniformly)in all directions. However, the microlens array 328 may align the lightemitted by each pixel to travel substantially anisotropically(non-uniformly) in one direction or in a narrow range of directions(e.g., an outgoing beam may spread or converge/focus by a small angle).In fact, it may be desirable in some cases. For example, the pixel 532may emit rays of light in all directions, but after the rays of lightreach the microlens 536, the rays of light may be all caused to travelin one direction. As shown, the rays of light emitted by pixel 532 mayall travel in parallel toward the eye 204 after they have passed throughthe microlens 536. As a result, the display 324 and microlens array 328are operable to create a light field using rays of light to simulate theappearance of an object.

The direction that the rays of light travel may depend on the locationof the emitting pixel relative to a microlens. For example, while therays emitted by the pixel 532 may travel toward the upper rightdirection, rays emitted by the pixel 522 may travel toward the lowerright direction because pixel 522 is located higher than pixel 532relative to their corresponding microlenses. Accordingly, the rays oflight for each pixel in pixel group may not necessarily travel towardthe eye. For example, the dotted rays of light emitted by pixel 524 maynot travel toward the eye 204 when the eye 204 is positioned as shown.

It should be appreciated that the display 324 may include rows andcolumns of pixels such that a pixel that is located into or out of thepage may generate rays of light that may travel into or out of the page.Accordingly, such light may be caused to travel in one direction into orout of the page after passing through a microlens.

It should also be appreciated that the display 324 may display an imagethat is recognizable or in focus only when viewed through the microlensarray 328. For example, if the image produced by the display 324 isviewed without the microlens array 328, it may not be equivalent to theimage perceived by the eye 204 with the aid of the microlens array 328even if viewed at a distance farther than the near plane 216. Thedisplay 324 may display a pre-filtered image, corresponding to a targetimage to be ultimately projected, that is unrecognizable when viewedwithout the microlens array 328. When the pre-filtered image is viewedwith the microlens array 328, the target image may be produced andrecognizable. A computer system or graphics processing system maygenerate the pre-filtered image corresponding to the target image.

It should further be noted that separate microlens arrays and/ordisplays may be placed in front of each eye of a viewer. Accordingly,binocular viewing may be achieved. As a result, the depth perceptioncues of binocular disparity and convergence may be fully orapproximately simulated. Each light field may also support the depth cueto accommodation (focusing) to be correctly simulated. Furthermore, byusing a pair of near-eye light field displays, binocular disparity,convergence, and accommodation are simultaneously and fully orapproximately simulated, producing a “comfortable” sensation of the 3Dscene extending behind the display 324.

In addition, since the synthesized light field may extend beyond thelens/pupil 208, the viewer may move left/right/up/down, rotate theirhead, or change the distance between their eye 204 (e.g., due todifferent users), maintaining the illusion of the virtual 3D scene.Embodiments of the present invention also support a fourth depth cuecalled motion parallax.

Further, it should be appreciated that microlens arrays and/or displaysmay occupy only a portion of the view of an observer.

Near-Eye Parallax Barrier Displays

FIG. 6A illustrates a side view of a near-eye parallax barrier display600, according to embodiments of the present invention. FIG. 6A includesthe eye 204 with the lens 208, retina plane 212, plane of focus 214, andnear plane 216 of FIG. 2. FIG. 6A also includes a display 624 and aspatial light modulator (SLM) array 626 (or a parallax barrier orpinhole array). An SLM may absorb or attenuate rays or light, withoutsignificantly altering their direction. Thus, an SLM may alter theintensity and possibly the color of a ray, but not its direction. SLMsmay include printed films, LCDs, light valves, or other mechanisms.

While the display 624 and SLM array 626 are within the minimumaccommodation distance 218, they are operable to produce a light fieldto simulate an object, in focus, from within the accommodation range ofthe eye 204. For example, a ray of light 621 may be produced by thedisplay 624 and SLM array 626 that is part of a light field simulatingan object that is located beyond the near plane 216.

Regions of the display 624 and SLM array 626 may be operable to switchbetween being transparent, semi-transparent, and/or opaque. As a result,rays of light that originate from beyond the display 624 and SLM array626 (e.g., from the surrounding environment) may still reach the eye204. For example, a ray of light 622 originating from the surface of anobject that may be 10 feet away may travel through the display 624 andSLM array 626 and to the eye 204. As a result, an observer may still beable to view at least portions of the surrounding environment.

FIG. 6B illustrates a side view of a near-eye parallax barrier displayand a microlens array, according to embodiments of the presentinvention. FIG. 6B includes similar elements as FIG. 3B. FIG. 6A alsoincludes a microlens array 328 b that may be disposed between the nearplane 216 and the display 324. The microlens array 328 b, may forexample, compress concave lenses rather than convex lenses. Thecombination of the microlens arrays 328 and 328 b may allow a ray 622 topass through a microlens system. The microlens arrays 328 and 328 b maycomprise a plurality of microlenses, in addition to other elementsincluding masks, prisms, or birefringent materials.

FIG. 7 illustrates a magnified side view of the near-eye parallaxbarrier display 600, according to embodiments of the present invention.FIG. 7 includes the display 624 and SLM array 626 of FIG. 6A. Thedisplay 624 may include multiple pixels, for example, pixels 722 and725. The SLM array 626 may include multiple pinholes operable to allow,block, or otherwise modulate the passage of light at various points ofthe SLM array 626, for example, pixels 730, 735, 740, and 745. Theparallax barrier 626 may be implemented with any spatial lightmodulator. For example, the parallax barrier 626 may be an LCD or OLED.

In one or more embodiments, the display 624 may include an array oflight-emitting elements (e.g., a semitransparent OLED) and the SLM array626 may include light-attenuating elements (e.g., a semitransparentLCD). In such an embodiment, rays of light 736, 741, and 746 originatingfrom the surrounding environment may not be modified by the display 624and the SLM array 626. Instead, modification to such rays of light maybe achieved using an additional light shutter that blocks the rays fromentering when the display 624 and the SLM array 626 are operating.

In one or more embodiments, both the display 624 and the SLM array 626are light-attenuating SLMs. One of the display 624 or the SLM array 626may display an array of slits/pinholes, while the other element displaysa pre-filtering image to synthesize a light field by attenuating rays oflight 736, 741, and 746 originating from the surrounding environmentthat pass through the layers. This would support “low power” caseswhere, by looking at a scene, rays are blocked to create text or images,rather than being emitted from the display 624, then blocked by the SLMarray 626.

The SLM array 626 may allow certain rays of light through while blockingother rays of light. For example, the pixel 730 may block a ray of light723 emitted by the pixel 722, while allowing the passage of another rayof light 724 emitted by the pixel 722. Accordingly, a light field may beproduced because the SLM array 626 causes the light to travelanisotropically in one direction. Alternatively, multiple rays of lightemitted by the pixel 722 may pass through the SLM array 626. In aconventional parallax barrier (slits and pinholes), only a singledirection may pass, but in a generalized solution multiple directionsmay pass (even all directions in some cases, resulting in no blocks ormodulation of rays of light emitted by the pixel 722). Further, the SLMarray 626 may partially attenuate light at varying degrees. For example,the pixel 745 may partially attenuate a ray of light 726 emitted by thepixel 725.

The display 624 may be a semi-transparent display (e.g., a transparentLCD or OLED). Accordingly, rays of light originating from behind boththe display 624 and the SLM array 626 from the perspective of the eye204 may be allowed to pass through the display 624. As a result, the eye204 may be able to view the surrounding environment even while thedisplay 624 and SLM array 626 are placed in front of the eye 204.

However, the SLM array 626 may allow or block such rays of lightoriginating from the surrounding environment. For example, a ray oflight 736 originating from the surrounding environment may be allowed topass through to the eye 204 by the pixel 735, while a ray of light 741originating from the surrounding environment may be blocked from passingthrough to the eye 204 by the pixel 740. The rays of light 736, 741, and746 may also be modulated by the display 624. Thus, the display 624 maybehave as another SLM similar to the SLM array 626, a semi-transparentlight emitter, or a combination of an SLM array and an emitter.

In addition, the SLM array 626 may partially attenuate such light atvarying degrees. For example, the pixel 745 may partially attenuate aray of light 746 originating from behind both the display 624 and theSLM array 626 from the perspective of the eye 204.

Accordingly, since rays of light from the surrounding environment mayreach the eye 204, a viewer may be able to generally view theenvironment while the display 624 and SLM array 626 may modify what theviewer can see by adding and/or removing rays of light. For example, alight-attenuating element (e.g., an LCD) may include black text in anobserver's view by blocking light, or a light-emitting element (e.g., anOLED) may include white text in an observer's view by emitting light. Asa result, the display 624 and SLM array 626 may provide an augmentedreality experience.

For example, FIG. 10 depicts a view through the near-eye parallaxbarrier display 600, according to embodiments of the present invention.The view includes the surrounding environment, which in this exampleincludes streets, buildings, trees, and so on. The near-eye parallaxbarrier display 600 may modify the view by including, for example, acoffee sign 1005 with an arrow 1010 pointing in the direction of a café.

In one or more embodiments of the invention, accommodation cues may beprovided. For example, if the arrow 1010 was instead labeling andpointing to the house 1015, and the viewer's eyes are focused on a car1020 that is located at a closer distance than the house 1015, the arrow1010 may be blurred slightly to approximate the same blurring amount ofthe house 1015. Accordingly, the natural human accommodation/defocuseffect may be simulated.

It should be appreciated that the near-eye parallax barrier display 600may provide a virtual reality experience when operating as an immersivedisplay, for example, by blocking all light from the surroundingenvironment and providing imagery through the display 624 and SLM array626.

In FIGS. 6 and 7, the SLM array 626 is between the eye 204 and thedisplay 624. However, it should be borne in mind that embodiments of theinvention allow for the display 624 to be between the eye 204 and theSLM array 626.

It should also be appreciated that the display 624 and/or SLM array 626may produce an image that is recognizable or in focus only when viewedwhile located closer than the near plane 216. For example, the image mayseem blurry or out of focus when viewed in the accommodation range. Thedisplay 624 may display a pre-filtered image, corresponding to a targetimage to be ultimately projected, that is unrecognizable when viewedwithout the SLM array 626. When the pre-filtered image is viewed withthe SLM array 626, the target image may be produced and recognizable. Acomputer system or graphics processing system may generate thepre-filtered image corresponding to the target image.

In addition, it should be borne in mind that FIGS. 6 and 7 illustratethe near-eye parallax barrier display 600 from a side view and that thenear-eye parallax barrier display 600 may be a three dimensional objectthat extends into or out of the page. For example, the near-eye parallaxbarrier display 600 may extend horizontally and vertically acrossreading glasses. It should further be noted that separate near-eyeparallax barrier displays may be placed in front of each eye of aviewer. In addition, it should be appreciated that the near-eye parallaxbarrier display 600 may occupy only a portion of the view of anobserver.

FIG. 8 illustrates a side view of a near-eye multilayer SLM display 800,according to embodiments of the present invention. The near-eyemultilayer SLM display 800 of FIG. 8 may be similar to the near-eyeparallax barrier display 600 of FIG. 6A. However, the near-eyemultilayer SLM display 800 of FIG. 8 includes multiple SLM arrays 826.By using multiple SLM arrays, the brightness, resolution, and/or thedepth of field may be improved. Further, by using high-speed SLMs thatrefresh faster than the human flicker fusion threshold, the resolutioncan approach that of the native display resolution. Embodiments of theinvention provide for the application of high-speed displays, as inFIGS. 6 and 7, to two-layer SLMS, other two-layer configurations, andmultilayer SLMs.

FIG. 9 illustrates a magnified side view of the near-eye multilayer SLMdisplay 800, according to embodiments of the present invention. FIG. 9is similar to FIG. 7 in that it includes the eye 204 and a display 824.However, FIG. 9 also includes multiple SLM arrays 826, for example, theSLM arrays 830, 832, and 834. In the embodiment shown, the multiple SLMarrays 826 include three SLM arrays. However, embodiments of theinvention allow for any number of SLM arrays.

The multiple SLM arrays 826 allow for increased control over the lightthat is allowed to pass through to the eye 204. For example, themultiple SLM arrays 826 may allow a more defined light field to beprovided to the eye 204 because each additional SLM array may help tofurther define the rays of light. As a result, the resolution and/ordepth of field of imagery may be improved. For example, a ray of light905 may be allowed to pass through to the eye 204 while a ray of light920 may be blocked by the SLM array 832, but would have otherwise beenable to pass if only the SLM array 830 was located between the ray oflight 920 and the eye 204. It should be appreciated that pixels in themultiple SLM arrays 826 may partially attenuate a ray of light, similarto the pixel 745 of FIG. 7.

Further, because the paths of multiple rays of light may overlap, suchrays may travel through the same SLM element of a SLM array, and as aresult, more light may be allowed to reach the eye 204. For example, therays of light 905 and 910 may travel through the same SLM element of theSLM array 832, and the rays of light 905 and 915 may travel through thesame SLM element of the SLM array 834.

In addition, the resolution or brightness may be increased by modulatingthe SLM arrays at high speeds. For example, if the human eye may be onlyable to detect images at 60 Hz, the SLM arrays may modulate ten timesfaster at 600 Hz. While a ray of light was blocked from traveling to theeye 204 during a first frame, the SLM arrays may modulate to allow thesame ray of light to pass through, thereby increasing resolution orbrightness.

In FIGS. 8 and 9, the multiple SLM arrays 826 are between the eye 204and the display 824. However, it should be borne in mind thatembodiments of the invention allow for the display 824 to be between theeye 204 and the multiple SLM arrays 826.

It should further be noted that separate SLM arrays and/or displays maybe placed in front of each eye of a viewer. Accordingly, binocularviewing may be achieved. As a result, the depth perception cues ofbinocular disparity and convergence may be fully or approximatelysimulated. Each light field may also support the depth cue toaccommodation (focusing) to be correctly simulated. Furthermore, byusing a pair of SLM arrays displays, binocular disparity, convergence,and accommodation are simultaneously and fully or approximatelysimulated, producing a “comfortable” sensation of the 3D scene extendingbehind the display 624 or 824.

In addition, since the synthesized light field may extend beyond thelens/pupil 208, the viewer may move left/right/up/down, rotate theirhead, or change the distance between their eye 204 (e.g., due todifferent users), maintaining the illusion of the virtual 3D scene.Embodiments of the present invention also support a fourth depth cuecalled motion parallax.

Further, it should be appreciated that SLM arrays and/or displays mayoccupy only a portion of the view of an observer.

Near-Eye Optical Deconvolution Displays

FIG. 11 illustrates a side view of a near-eye optical deconvolutiondisplay 1100, according to embodiments of the present invention. FIG. 11includes the eye 204 with the lens 208, retina plane 212, plane of focus214, and near plane 216 of FIG. 2. FIG. 11 also includes a first display1124 and optionally additional displays like display 1125. Thesedisplays may be located nearer to the eye 204 than the near plane 216.Therefore, as discussed with relation to FIG. 2A, an image displayed bythe display 1124 will be typically out of focus to the eye 204.

However, embodiments of the present invention allow for the display 1124to produce an image that is clear and in focus when perceived by the eye204. Surfaces viewed at such close distances are blurred in a certainway. Embodiments of the invention allow for the display of an image thathas been inversely blurred so that a natural blurring effect of an eyewill cancel out the inverse blur, resulting in an in focus image.

FIG. 12A depicts images before and after convolution, according toembodiments of the present invention. FIG. 12A includes a dot 1204 on asurface. When the dot 1204 is viewed by an eye within the minimumaccommodation distance of the eye, the dot 1204 may appear blurred to anobserver. For example, the perceived blurred image may be depicted by adisk 1208. A function s(x, y) describing the disk 1208 may be the resultof a convolution operation of a function i(x, y) describing the dot 1204with a second function h(x, y). The second function may be, for example,the point spread function (PSF). The point spread function may describethe effect of a defocused eye attempting to view a plane outside theaccommodation distance of the eye.

Accordingly, the natural blurring effect caused by the eye may bedescribed by a convolution operation. For example, the followingmathematical equation may describe the relationship between the dot 1204and the disk 1208:i(x,y)*h(x,y)=s(x,y)

FIG. 12B depicts images before and after deconvolution, according toembodiments of the present invention. FIG. 12B includes the same dot1204 as in FIG. 12A. In order to cancel, reverse, or counter theblurring effect caused by the eye, a deconvolved or pre-filtered imagemay be produced. For example, a deconvolved dot 1212 of the dot 1204 maybe produced by performing a deconvolution operation on the dot 1204. Theresult of the deconvolution operation, e.g., the deconvolved dot 1212,may be depicted by two concentric rings. The two concentric rings mayhave differing intensities.

More specifically, if the dot 1204 described by the function i(x, y) isconvoluted with the inverse of the second function h⁻¹(x, y), theresulting function describing the deconvolved dot 1212 may be ĩ(x, y).The inverse of the second function may be, for example, the inverse ofthe PSF.

Accordingly, the opposite or inverse of the natural blurring effectcaused by the eye may be described by a deconvolution operation. Thefollowing mathematical equation may describe the relationship betweenthe dot 1204 and the deconvolved dot 1212:i(x,y)*h ⁻¹(x,y)={tilde over (i)}(x,y)

The deconvolution operation may reduce in negative values, which may notbe synthesized by the display or values outside the dynamic range of thedisplay. The deconvolved image ĩ(x, y) may be filtered to transform thedeconvolution output to be within the dynamic range of the displaydevice.

FIG. 12C depicts a deconvolved image before and after convolution,according to embodiments of the present invention. When a convolutionoperation is performed on a function describing a deconvolved image, theresulting function may describe the original image. For example, whenthe deconvolved dot 1212 described by ĩ(x, y) undergoes a convolutionoperation with the second function h(x, y), the result may be thefunction i(x, y) describing the original dot 1204. The second functionmay be, for example, the PSF.

The following mathematical equation may describe the relationshipbetween the deconvolved dot 1212 and the dot 1204:{tilde over (i)}(x,y)*h(x,y)=i(x,y)

Accordingly, an eye may perceive an image completely or at leastapproximately similar to the original dot 1204 in focus when viewing adeconvolved version 1212 of the dot in a near-eye range (nearer to theeye than the near plane of the eye) because the eye's convolution effectmay translate the deconvolved version of the dot completely or at leastapproximately similar to the original dot 1204. This approximation mayhave reduced contrast or other artifacts, but may still improve thelegibility or recognizability of the image, as compared to aconventional display without pre-filtering or deconvolution applied.

It should be appreciated that the function i(x, y) may describe multiplepoints or pixels on a surface that together form an image. Accordingly,the deconvolved function ĩ(x, y) may correspond to multiple points orpixels that together form a deconvolved version of the image. As aresult, when the deconvolved version of the image described by thedeconvolved function ĩ(x, y) is viewed in near-eye ranges, the originalimage described by the function i(x, y) may be perceived by an observer.

Returning to FIG. 11, a deconvolved image may be displayed by thedisplay 1124. Since the display 1124 is within the near-eye range, theobserver may perceive a convoluted version of the deconvolved image. Asdiscussed above, a convolution of an image deconvolved by the inverse ofthe convolution function will result in substantially the originalimage. Accordingly, the observer will perceive an in focus image sincethe blurring effect of the eye will have been countered by the displayof the deconvolved image. Therefore, an image may be recognizable by anobserver in near-eye ranges.

It should be appreciated that embodiments of the present invention allowfor pre-filtering processes other than deconvolution. For example, otheroperations besides deconvolution may be used to create a pre-filteredimage that when viewed at near-eye distances, provides a recognizableimage to an observer after undergoing the eye's convolution effect.

It should be appreciated that multiple displays may be used. It shouldfurther be appreciated that the displays 1124 and 1125 may besemi-transparent. As a result, the eye 204 may be able to view imagesdisplayed by the display 1124 through the display 1125. The eye 204 mayalso be able to view the surrounding environment through both thedisplays 1124 and 1125. Multiple layers of displays may also decrease oreliminate artifact ringing and improve contrast.

It should also be appreciated that optical deconvolution displays mayblock the light from the surrounding environment to provide VRapplications. For example, a display may block a portion of anobserver's view while providing a deconvolved image in another portion.Or, for example, a first display in a multilayer deconvolution displaymay block light while a second display provides a deconvolved image.

Alternatively, such displays may generally allow the light from thesurrounding environment and block only portions of the incoming lightand/or augment portions with light produced by the display to provide ARapplications.

It should also be appreciated that the displays 1124 and 1125 maydisplay an image that is recognizable or in focus only when viewed whilelocated closer than the near plane 216. For example, the image may seemblurry or out of focus when viewed in the accommodation range. Thedisplays 1124 and 1125 may display a pre-filtered image, correspondingto a target image to be ultimately projected, that is unrecognizablewhen viewed within the accommodation range. When the pre-filtered imageis viewed within the accommodation range, the target image may berecognizable. A computer system or graphics processing system maygenerate the pre-filtered image corresponding to the target image.

Additional Embodiments

It should be appreciated that embodiments of the invention provide forcombining layers of near-eye light field displays, near-eye parallaxbarrier displays, and/or near-eye optical deconvolution displays. Lightfield displays and optical deconvolution displays may present differentperformance trade-offs. Light field displays may require high-resolutionunderlying displays to achieve sharp imagery, but otherwise preserveimage contrast. In contrast, optical deconvolution displays may preserveimage resolution, but reduce contrast.

The light field displays and optical deconvolution displays may becombined in order to benefit from the performance of each display and tosupport a continuous trade-off between resolution and contrast. Forexample, embodiments of the invention support performing opticaldeconvolution in the light field domain, rather than appliedindependently to each display layer.

Near-eye light field displays, near-eye parallax barrier displays,and/or near-eye optical deconvolution displays may be combined becausesuch displays may implement semi-transparent displays. For example, suchdisplays may implement a combination of light-attenuating (e.g., LCD) orlight-emitting (e.g., OLED) displays.

It should be appreciated that embodiments of the invention allow for theuse of multiple displays tiled together to form one effective display.For example, the display 324, display 624, display 824, or display 1124and 1125 may comprise multiple sub-displays. Sub-displays may be tiled,e.g. side by side, to synthesize a form display. Unlike multiple monitorworkstations, any gaps between displays may not introduce artifactsbecause the pre-filtered images may be modified to display on each tileto accommodate for the gaps between them.

Embodiments of the invention provide for both virtual reality (VR) andaugmented reality (AR) applications. For example, near-eye light fielddisplays, near-eye parallax barrier displays, and/or near-eye opticaldeconvolution displays may block the light from the surroundingenvironment to provide VR applications. Alternatively, such displays maygenerally allow the light from the surrounding environment and blockonly portions of the incoming light and/or augment portions with lightproduced by the display to provide AR applications.

In various embodiments, light from the surrounding environment mayfunction as a backlight, with the display layers attenuating theincident light field. In some embodiments, at least one display layermay contain light-emitting elements (e.g., an OLED panel). Inembodiments of the invention, a combination of light-attenuating andlight-emitting layers can be employed. It should be appreciated thatmore than one layer may emit light. For example, in FIG. 9, in additionto display 824, SLM arrays 830, 832, and 834 may also emit light.

In one or more embodiments, each display layer may include either alight-attenuating display or a light-emitting display, or a combinationof both (each pixel may attenuate and/or emit rays of light). Furtherembodiments may include multilayer devices, for example, OLED and LCD,LCD and LCD, or and so on.

For near-eye light field displays for VR applications, a 2D display maybe covered with either a parallax barrier or microlens array to supportcomfortable accommodation. Furthermore, multiple light-attenuatinglayers may be used to increase brightness, resolution, and depth offield.

Further embodiments of the invention may include holographic displayelements. For example, as the resolution increases, the pitch may becomesmall enough such that diffractive effects may be accounted for. Imageformation models and optimization methods may be employed to account fordiffraction, encompassing the use of computer-generated holograms fornear-eye displays in a manner akin to light field displays. Embodimentsof the present invention provide for applying optical deconvolution toholographic systems, thereby eliminating the contrast loss observed withincoherent displays.

Embodiments of the present invention provide for lightweight“sunglasses-like” form factors with a wide field of view using near-eyedisplays as discussed above. Such displays can be practicallyconstructed at high volumes and at low cost. Such displays may have aviable commercial potential as information displays, for example,depicting basic status messages, the time of day, and augmenting thedirectly perceived physical world.

Embodiments of the present invention provide for adjusting producedimages to account for aberrations or defects of an observer's eyes. Theaberrations may include, for example, myopia, hyperopia, astigmatism,and/or presbyopia. For example, a near-eye light field display, near-eyeparallax display, or near-eye optical deconvolution display may produceimages to counteract the effects of the observer's aberrations based onthe observer's optical prescription. As a result, an observer may beable to view images in focus without corrective eyewear like eyeglassesor contact lenses. It should be appreciated that embodiments of theinvention may also automatically calibrate the vision correctionadjustments with the use of a feedback system that may determine thedefects of an eye.

Embodiments of the invention may also adjust the provided image based oninformation from an eye-track adjustment system that may determine thedirection of gaze and/or the distance of the eye from the display(s).Accordingly, the display(s) may adjust the image displayed to optimizethe recognizability of the image for different directions of gaze,distances of the eye from the display, and/or aberrations of the eye.

Embodiments of the invention may also adjust the provided image based oninformation from one or more sensors. For example, embodiments mayinclude an environmental motion-tracking component that may include acamera. The environmental motion-tracking component may track movementor changes in the surrounding environment (e.g., movement of objects orchanges in lighting). In a further example, the movement of a user'sbody may be tracked and related information may be provided. As aresult, embodiments of the invention may adjust the provided image basedon the environment of a user, motions of a user, or movement of a user.

In another example, embodiments of the invention may include an internalmotion-tracking component that may include a gyroscopic sensor,accelerometer sensor, an electronic compass sensor, or the like. Theinternal motion-tracking component may track movement of the user andprovide information associated with the tracked movement. As a result,embodiments of the invention may adjust the provided image based on themotion. In other examples, sensors may determine and provide thelocation of a user (e.g., GPS), a head position or orientation of auser, the velocity and acceleration of the viewer's head position andorientation, environmental humidity, environmental temperature,altitude, and so on.

Information related to the sensor determinations may be expressed ineither a relative or absolute frame of reference. For example, GPS mayhave an absolute frame of reference to the Earth's longitude andlatitude. Alternatively, inertial sensors may have a relative frame ofreference while measuring velocity and acceleration relative to aninitial state (e.g., the phone is currently moving a 2 mm per second vs.the phone is at a given latitude/longitude).

Near-eye light field displays, near-eye parallax barrier displays,and/or near-eye optical deconvolution displays may be included ineyeglasses. For example, such displays may replace conventional lensesin a pair of eyeglasses.

FIG. 13 depicts a flowchart 1300 of an exemplary process of displaying anear-eye image, according to an embodiment of the present invention. Ina block 1302, a pre-filtered image to be displayed is determined,wherein the pre-filtered image corresponds to a target image. Forexample, a computer system may determine a pre-filtered image that maybe blurry when viewed by itself in an accommodation range but in focuswhen viewed through a filter or light field generating element.

In a block 1304, the pre-filtered image is displayed on a display. Forexample, in FIGS. 3B, 6, and 8, a pre-filtered image is displayed on thedisplay 324, 624, and 826, respectively.

In a block 1306, a near-eye light field is produced after thepre-filtered image travels through a light field generating elementadjacent to the display, wherein the near-eye light field is operable tosimulate a light field corresponding to the target image. For example,in FIG. 3A, a light field corresponding to a target image is producedafter the pre-filtered image passes through the microlens array 328.Similarly, in FIGS. 6 and 8, a light field corresponding to a targetimage is produced after the pre-filtered image passes through the SLMarray 626 and multiple SLM arrays 826, respectively.

FIG. 14 depicts a flowchart 1400 of an exemplary process of displaying anear-eye image, according to an embodiment of the present invention. Ina block 1402, a target image is received. For example, a computer systemmay receive a target image from a graphics processing system

In a block 1404, a deconvolved image corresponding to a target image isdetermined, wherein when the deconvolved image is displayed within anear-eye range of an observer, the target image may be perceived infocus by the observer. For example, in FIG. 12B, a deconvolved versionof a target image is determined. As in FIG. 12C, when the deconvolvedversion of the target image undergoes a convolution operation of theeye, the target image is perceived in focus by an observer.

In a block 1406, the deconvolved image is displayed on a display. Forexample, in FIG. 11, a deconvolved image may be displayed on a display1124 or 1125.

It should be appreciated that while embodiments of the present inventionhave been discussed and illustrated with various displays located withinthe near-plane but a distance from the eye, for example in FIGS. 3B, 6,8, 11, embodiments of the present invention also provide for displaysadjacent to the eye. For example, one or more layers of displays may beoperable to adjoin an eye, similar to a contact lens. Because suchdisplays may have a semi-spherical shape, the displays may account foraffects of the shape to provide a sharp and recognizable image to theeye.

While the foregoing disclosure sets forth various embodiments usingspecific block diagrams, flowcharts, and examples, each block diagramcomponent, flowchart step, operation, and/or component described and/orillustrated herein may be implemented, individually and/or collectively,using a wide range of hardware, software, or firmware (or anycombination thereof) configurations. In addition, any disclosure ofcomponents contained within other components should be considered asexamples because many other architectures can be implemented to achievethe same functionality.

The process parameters and sequence of steps described and/orillustrated herein are given by way of example only. For example, whilethe steps illustrated and/or described herein may be shown or discussedin a particular order, these steps do not necessarily need to beperformed in the order illustrated or discussed. The various examplemethods described and/or illustrated herein may also omit one or more ofthe steps described or illustrated herein or include additional steps inaddition to those disclosed.

While various embodiments have been described and/or illustrated hereinin the context of fully functional computing systems, one or more ofthese example embodiments may be distributed as a program product in avariety of forms, regardless of the particular type of computer-readablemedia used to actually carry out the distribution. The embodimentsdisclosed herein may also be implemented using software modules thatperform certain tasks. These software modules may include script, batch,or other executable files that may be stored on a computer-readablestorage medium or in a computing system. These software modules mayconfigure a computing system to perform one or more of the exampleembodiments disclosed herein. One or more of the software modulesdisclosed herein may be implemented in a cloud computing environment.Cloud computing environments may provide various services andapplications via the Internet. These cloud-based services (e.g.,software as a service, platform as a service, infrastructure as aservice, etc.) may be accessible through a Web browser or other remoteinterface. Various functions described herein may be provided through aremote desktop environment or any other cloud-based computingenvironment.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as may be suited to theparticular use contemplated.

Embodiments according to the invention are thus described. While thepresent disclosure has been described in particular embodiments, itshould be appreciated that the invention should not be construed aslimited by such embodiments, but rather construed according to the belowclaims.

What is claimed is:
 1. An apparatus comprising: a display comprising aplurality of pixels, said plurality of pixels comprising a first groupof pixels and a second group of pixels operable to isotropically andomnidirectionally emit a first plurality of rays and a second pluralityof rays, respectively; a computer system coupled with said display andoperable to instruct said display to display images; and a microlensarray comprising a plurality of microlenses operable to switch between aplurality of operating modes, said micro lens array being locatedadjacent to said display and, operable to produce a light field displayby altering light emitted by said display to simulate an object that isin focus to an observer while said display and said microlens array arelocated within a near-eye range of said observer, wherein said firstplurality of rays is received by a first microlens of said microlensarray and aligned by said first microlens in a plurality of firstdirections in parallel based on a location of said first group of pixelsrelative to said first microlens, further wherein said second pluralityof rays is received by a second microlens of said microlens array andaligned by said second microlens in a plurality of second directions inparallel based on a location of said second group of pixels relative tosaid second microlens.
 2. The apparatus of claim 1, further comprising aplurality of groups of pixels in said plurality of pixels, wherein eachmicrolens of said plurality of microlenses corresponds to a group ofpixels in said plurality of groups of pixels.
 3. The apparatus of claim2, wherein each microlens is operable to alter light from each pixel ina group of pixels corresponding to said each microlens.
 4. The apparatusof claim 1, wherein said microlens array is operable to projectanisotropic light by altering isotropic light produced by said display.5. The apparatus of claim 1, wherein said light field is operable tosimulate a 3D object located beyond said near-eye range of saidobserver.
 6. The apparatus of claim 1, wherein said computer system isoperable to determine an image for display that counteracts aberrationsof said observer's eye.
 7. The apparatus of claim 1, further comprisinga feedback system operable to make measurements of aberrations of saidobserver's eye; and wherein said computer system is further operable todetermine an image for display that counteracts said aberrations basedon said measurements.
 8. The apparatus of claim 1, further comprising: asensor operable to provide information related to a surroundingenvironment; and wherein said computer system is further operable todetermine an image for display that counteracts said aberrations basedon said information.
 9. The apparatus of claim 1, further comprising aneye-track adjustment system operable to track a gaze of an eye, whereinsaid eye-track adjustment system is operable to communicate informationrelated to a gaze of an eye to said computer system for determination ofan image for display by said computer system based on said information.10. The apparatus of claim 1, wherein said plurality of microlenses areoperable to electro-optically switch between a transparent and opaquestate.
 11. The apparatus of claim 1, wherein said display comprises aplurality of sub-displays disposed side by side to one another.
 12. Theapparatus of claim 1, wherein said microlens array comprises at leastone microlens from the group consisting of: a liquid crystal microlens;and a birefringent optics microlens.
 13. The apparatus of claim 1,wherein said microlens array is configured to alternate between atransparent mode and an opaque mode.
 14. The apparatus of claim 1,wherein said microlens array is further configured to switch betweensaid plurality of operating modes comprising a transparent mode and aopaque mode, said plurality of operating modes comprising at least oneoperating mode from the group of operating modes consisting of: aspatial-multiplexing of said transparent mode and said opaque mode; aspatial-combination of said transparent mode and said opaque mode; and atemporal modulation of said transparent mode and said opaque mode. 15.The apparatus of claim 1, wherein said apparatus comprises a pluralityof separate microlens arrays in front of at least one eye of saidobserver operable to achieve binocular viewing for said observer. 16.The apparatus of claim 1, wherein said apparatus comprises a pluralityof displays in front of at least one eye of said observer operable toachieve binocular viewing for said observer.
 17. An apparatuscomprising: a display operable to produce an image and comprising aplurality of pixels, said plurality of pixels operable to emit aplurality of rays isotropically and omnidirectionally; and a microlensarray located adjacent to said display and operable to switch between aplurality of operating modes, said microlens array together with saiddisplay being operable to produce a light field display simulating a 3Dobject that is recognizable to an observer while said display and saidmicrolens array are located within a near-eye range of said observer,wherein a first plurality of rays emitted from a first group of pixelsof said display is received by a first microlens of said microlens arrayand aligned by said first microlens in a plurality of first directionsin parallel based on a location of said first group of pixels relativeto said first microlens, wherein a second plurality of rays emitted froma second group of pixels of said display is received by a secondmicrolens of said microlens array and aligned by said second microlensin a plurality of second directions in parallel based on a location ofsaid second group of pixels relative to said second microlens.
 18. Theapparatus of claim 17, wherein said image is out of focus if viewedwithout said microlens array and said image is in focus if viewedthrough said microlens array.
 19. The apparatus of claim 17, whereinsaid microlens array is operable to display different versions, of animage produced by said display, in different directions.
 20. Theapparatus of claim 17, wherein said microlens array is operable to covera portion of a view of said observer less than said observer's entireview.
 21. The apparatus of claim 17, wherein said microlens array andsaid display are operable to provide a virtual reality experience. 22.The apparatus of claim 17, wherein said display is semi-transparent. 23.The apparatus of claim 17, wherein said display is an LCD or OLED.
 24. Amethod comprising: determining a pre-filtered image to be displayed,wherein said pre-filtered image corresponds to a target image;displaying said pre-filtered image on a display; and producing anear-eye light field after said pre-filtered image travels through amicrolens array operable to switch between a plurality of operatingmodes and adjacent to said display, wherein said near-eye light field isoperable to simulate a light field corresponding to said target image,wherein a first plurality of rays emitted isotropically andomnidirectionally from a first group of pixels of said display isreceived by a first microlens and aligned by said first microlens in aplurality of first directions in parallel based on a location of saidfirst group of pixels relative to said first microlens, wherein a secondplurality of rays emitted isotropically and omnidirectionally from asecond group of pixels of said display is received by a second microlensand aligned by said second microlens in a plurality of second directionsin parallel based on a location of said second group of pixels relativeto said second microlens, wherein said second group of pixels projectsrays to said second microlens.
 25. The method of claim 24, wherein saidtarget image simulated by said near-eye light field is in focus to anobserver's eye when said display and said microlens array are in anear-eye range of said observer's eye.
 26. The method of claim 24,wherein said microlens array is operable to project anisotropic light byaltering isotropic light produced by said display.
 27. The method ofclaim 24, wherein said determining is executed by a computer systembased on aberrations of said observer's eye, a gaze of said observer'seye, and a distance between said observer's eye and said microlensarray.
 28. The method of claim 24, wherein said microlens array isoperable to display different versions of said target image in differentdirections.
 29. The method of claim 24, wherein said microlens array andsaid display are operable to provide a virtual reality experience.